Building a Forecaster using AutoMLx

by the Oracle AutoMLx Team


Forecasting Demo notebook.

Copyright © 2024, Oracle and/or its affiliates.

Licensed under the Universal Permissive License v 1.0 as shown at https://oss.oracle.com/licenses/upl/

Overview of this Notebook¶

In this notebook we will build a forecaster using the Oracle AutoMLx tool for three real-world datasets. We explore the various options available in the Oracle AutoMLx Forecasting module, allowing the user to control the AutoML training process. We finally evaluate the forecasting algorithms using in-built visualization tools. Finally, we provide an overview of the capabilities that Oracle AutoMLx offers for explaining the predictions of the tuned model.


Prerequisites¶

  • Experience level: Novice (Python and Machine Learning)
  • Professional experience: Some industry experience

Business Use¶

Forecasting uses historical time series data as input to make informed estimates of future trends. Learning an accurate forecasting model requires expertise in data science and statistics. This process typically comprises of:

  • Preprocess dataset (clean, impute, engineer features, normalize).
  • Pick an appropriate model for the given dataset and prediction task at hand.
  • Tune the chosen model’s hyperparameters for the given dataset.

These steps are significantly time consuming and heavily rely on data scientist expertise. Unfortunately, to make this problem harder, the best feature subset, model, and hyperparameter choice widely varies with the dataset and the prediction task. Hence, there is no one-size-fits-all solution to achieve reasonably good model performance. Using a simple Python API, AutoML can quickly jump-start the datascience process with an accurately-tuned model and appropriate features for a given prediction task.

Table of Contents¶

  • Setup
  • Load the M4 Forecasting Competition dataset
  • Univariate time series (single-target forecasting)
    • Split data into train and test for the forecasting task
    • Setting the execution engine
    • Create an instance of Oracle AutoMLx
    • Train a forecasting model using AutoMLx
    • Generate and visualize forecasts
    • Analyze the AutoML optimization process
      • Algorithm Selection
      • Hyperparameter Tuning
  • Multivariate time series
    • Single-target Forecasting with Exogenous Variables
    • Multi-target Forecasting with Exogenous Variables
      • Advanced AutoML Configuration
      • Train a model using Oracle AutoMLx
      • Specify the number of cross-validation (CV) folds
      • Make predictions
      • Visualization
  • Machine Learning Explainability (MLX)
    • Initialize an MLExplainer
    • Prediction Explanations (Comparative Feature Importance)
  • References

Setup¶

Basic setup for the Notebook.

In [1]:
%matplotlib inline
%load_ext autoreload
%autoreload 2

Load the required modules.

In [2]:
import time
import datetime
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

try:
    from sktime.forecasting.model_selection import temporal_train_test_split
except ImportError:
    try:
        from sktime.split import TemporalTrainTestSplitter as temporal_train_test_split
    except ImportError as e:
        raise ImportError("Failed to import Splitters from sktime. "
                          "Please ensure you have the correct version of sktime installed.") from e

plt.rcParams['figure.figsize'] = [15, 5]
plt.rcParams['font.size'] = 15

import automlx
from automlx import init

Load the M4 Forecasting Competition dataset¶

We fetch the series from the repository of the M4 forecasting competition to use throughout this demo.

In [3]:
! wget https://github.com/Mcompetitions/M4-methods/raw/master/Dataset/Train/Weekly-train.csv -q
! wget https://github.com/Mcompetitions/M4-methods/raw/master/Dataset/M4-info.csv -q
In [4]:
all_series = pd.read_csv("Weekly-train.csv", index_col=0)  # consists of thousands of series
metadata_csv = pd.read_csv("M4-info.csv", index_col=0)  # describes their datetime index

Univariate time series (single-target forecasting)¶

The Oracle AutoMLx solution for forecasting can process both univariate (where only a single time series is available) and multivariate time series (where multiple time series are available). We start by displaying an example of use for univariate time series, and will address multivariate data at the end of this notebook.

We select one time series from the finance sector with weekly collection frequency. M4 dataset requires additional preprocessing to reconstruct the time series.

In [5]:
series_id = 'W142'
series_metadata = metadata_csv.loc[series_id]
series_values = all_series.loc[series_id]

# drop NaNs for the time period where data wasn't recorded
series_values.dropna(inplace=True)

# retrieve starting date of recording and series length to generate the datetime index
start_date = pd.to_datetime(series_metadata.StartingDate)
future_dates = pd.date_range(start=start_date,
                             periods=len(series_values),
                             freq='W')
y = pd.DataFrame(series_values.to_numpy(),
                 index=future_dates,
                 columns=[(series_metadata.category+"_"+series_id)])

We can now visualize the last 200 weeks of data we have on hand.

In [6]:
y = y.tail(n=200)  # approximately 4 years of data
y.plot(ylabel='Weekly Series '+series_id, grid=True)
Out[6]:
<Axes: ylabel='Weekly Series W142'>

One must ensure that the data points are in a Pandas DataFrame, sorted in chronological order.

In [7]:
print(y.index)
print("Time Index is", "" if y.index.is_monotonic_increasing else "NOT", "monotonic.")
print("Train datatype", type(y))
DatetimeIndex(['2012-12-09 12:00:00', '2012-12-16 12:00:00',
               '2012-12-23 12:00:00', '2012-12-30 12:00:00',
               '2013-01-06 12:00:00', '2013-01-13 12:00:00',
               '2013-01-20 12:00:00', '2013-01-27 12:00:00',
               '2013-02-03 12:00:00', '2013-02-10 12:00:00',
               ...
               '2016-07-31 12:00:00', '2016-08-07 12:00:00',
               '2016-08-14 12:00:00', '2016-08-21 12:00:00',
               '2016-08-28 12:00:00', '2016-09-04 12:00:00',
               '2016-09-11 12:00:00', '2016-09-18 12:00:00',
               '2016-09-25 12:00:00', '2016-10-02 12:00:00'],
              dtype='datetime64[ns]', length=200, freq='W-SUN')
Time Index is  monotonic.
Train datatype <class 'pandas.core.frame.DataFrame'>

Split data into train and test for the forecasting task¶

As can be seen above, the data contains 100 weekly recorded values over the past 4 years. We will try to predict electricity consumption for the last 0.5 year of data (26 data points), using the previous years as training data. Hence, we separate the dataset into training and testing sets using Temporal train-test split, which ensures that the continuity of the input time series is preserved. Each point in the series represents a month, so we will hold out the last 26 points as test data.

In [8]:
y_train, y_test = temporal_train_test_split(y, test_size=26)
print("Training length: ", len(y_train)," Testing length: ", len(y_test))
Training length:  174  Testing length:  26
In [9]:
print("y_train", y_train)
print("\ny_test", y_test)
y_train                      Finance_W142
2012-12-09 12:00:00      4793.269
2012-12-16 12:00:00      4818.969
2012-12-23 12:00:00      4863.783
2012-12-30 12:00:00      4926.357
2013-01-06 12:00:00      4916.616
...                           ...
2016-03-06 12:00:00      5024.759
2016-03-13 12:00:00      5021.153
2016-03-20 12:00:00      4996.524
2016-03-27 12:00:00      5004.032
2016-04-03 12:00:00      5053.536

[174 rows x 1 columns]

y_test                      Finance_W142
2016-04-10 12:00:00      5039.544
2016-04-17 12:00:00      5053.420
2016-04-24 12:00:00      5066.860
2016-05-01 12:00:00      5099.887
2016-05-08 12:00:00      5082.291
2016-05-15 12:00:00      5084.340
2016-05-22 12:00:00      5081.651
2016-05-29 12:00:00      5106.797
2016-06-05 12:00:00      5147.906
2016-06-12 12:00:00      5140.993
2016-06-19 12:00:00      5153.210
2016-06-26 12:00:00      5258.369
2016-07-03 12:00:00      5307.455
2016-07-10 12:00:00      5295.631
2016-07-17 12:00:00      5300.088
2016-07-24 12:00:00      5295.456
2016-07-31 12:00:00      5333.739
2016-08-07 12:00:00      5338.803
2016-08-14 12:00:00      5339.390
2016-08-21 12:00:00      5345.043
2016-08-28 12:00:00      5369.168
2016-09-04 12:00:00      5401.186
2016-09-11 12:00:00      5395.298
2016-09-18 12:00:00      5386.552
2016-09-25 12:00:00      5377.376
2016-10-02 12:00:00      5406.685

Setting the execution engine¶

The AutoML pipeline offers the function init, which allows to initialize the parallelization engine.

In [10]:
init(engine='ray')
[2024-08-15 19:16:09,958] [automlx.backend] Overwriting ray session directory to /tmp/xppf3m98/ray, which will be deleted at engine shutdown. If you wish to retain ray logs, provide _temp_dir in ray_setup dict of engine_opts when initializing the AutoMLx engine.

Create an instance of Oracle AutoMLx¶

The Oracle AutoMLx solution automatically provides a tuned forecasting pipeline that best models the given training dataset and a prediction task at hand. Here the dataset can be any univariate time-series.

AutoML for Forecasting consists of three main modules:

  • Preprocessing
    • Impute any missing values using back fill or forward fill mechanisms to ensure input has a well-defined and consistent frequency.
    • Identify seasonalities present in the data by detrending and analyzing the Autocorrelation Function (ACF) of the series.
    • Decide appropriate number of cross-validation (CV) folds and the forecast horizons based on the datetime frequency of data.
  • Algorithm Selection: Identify the right algorithm for a given dataset, choosing from the following:
    • NaiveForecaster - Naive and Seasonal Naive method
    • ThetaForecaster - Equivalent to Simple Exponential Smoothing (SES) with drift
    • ExpSmoothForecaster - Holt-Winters' damped method
    • STLwESForecaster - Seasonal Trend LOESS (locally weighted smoothing) with Exponential Smoothing substructure
    • STLwARIMAForecaster - Seasonal Trend LOESS (locally weighted smoothing) with ARIMA substructure
    • SARIMAXForecaster - Seasonal Autoregressive Integrated Moving Average
    • ETSForecaster - Error, Trend, Seasonality (ETS) Statespace Exponential Smoothing
    • ProphetForecaster (optional) - Facebook Prophet, only available when explicitly provided to the AutoML Pipeline through the model_list parameter.
    • VARMAXForecaster - Vector AutoRegressive Moving Average with Exogenous Variables (Available for multi-target forecasting)
    • DynFactorForecaster - Dynamic Factor Models in state-space form with Exogenous Variables (Available for multi-target forecasting)
    • ExtraTreesForecaster - A reduction-based machine learning model based on ExtraTrees regression model
    • LGBMForecaster - A reduction-based machine learning model based on LGBM regression model
    • XGBForecaster - A reduction-based machine learning model based on XGB regression model
  • Hyperparameter Tuning
    • Find the right model parameters that maximize score for the given dataset.

These pieces are readily combined into a simple AutoML pipeline which automates the entire forecasting process with minimal user input/interaction. One can then evaluate and visualize the forecast produced by the selected model, and optionally the other tuned models.

Train a forecasting model using Oracle AutoMLx¶

The AutoML API is quite simple to work with. We first create an instance of the pipeline. Next, the training data is passed to the fit() function which successively executes the previously mentioned modules.

The generated model can then be used for forecasting tasks. By default, we use the negative of symmetric mean absolute percentage error (sMAPE) scoring metric to evaluate the model performance

In [11]:
est1 = automlx.Pipeline(task='forecasting', n_algos_tuned=2)
est1.fit(X=None, y=y_train)

print('Selected model: {}'.format(est1.selected_model_))
print('Selected model params: {}'.format(est1.selected_model_params_))
[2024-08-15 19:16:14,515] [automlx.data_transform] Number of simple differencing orders required: d = 1
[2024-08-15 19:16:14,522] [automlx.data_transform] Seasonal Periodicities; from decomposed/adjusted: [52, 35, 1]
[2024-08-15 19:16:14,766] [automlx.interface] Dataset shape: (174,2)
[2024-08-15 19:16:15,056] [automlx.data_transform] Running preprocessing. Number of features: 3
[2024-08-15 19:16:15,260] [automlx.data_transform] Preprocessing completed. Took 0.204 secs
[2024-08-15 19:16:15,470] [automlx.process] Running Model Generation
[2024-08-15 19:16:15,602] [automlx] Provided model (VARMAXForecaster) is not supported.Supported models are: ['LGBMForecaster', 'ExtraTreesForecaster', 'ExpSmoothForecaster', 'NaiveForecaster', 'SARIMAXForecaster', 'STLwARIMAForecaster', 'ThetaForecaster', 'ProphetForecaster', 'ETSForecaster', 'STLwESForecaster', 'XGBForecaster']
[2024-08-15 19:16:15,603] [automlx] Provided model (DynFactorForecaster) is not supported.Supported models are: ['LGBMForecaster', 'ExtraTreesForecaster', 'ExpSmoothForecaster', 'NaiveForecaster', 'SARIMAXForecaster', 'STLwARIMAForecaster', 'ThetaForecaster', 'ProphetForecaster', 'ETSForecaster', 'STLwESForecaster', 'XGBForecaster']
[2024-08-15 19:16:15,614] [automlx.process] Model Generation completed.
[2024-08-15 19:16:15,702] [automlx.model_selection] Running Model Selection
[2024-08-15 19:16:25,389] [automlx.dataset] Forecast horizon set to 13 for validation sets.
[2024-08-15 19:16:37,131] [automlx.model_selection] Model Selection completed - Took 21.430 sec - Selected models: ['ETSForecaster', 'STLwARIMAForecaster']
[2024-08-15 19:16:37,626] [automlx.trials] Running Model Tuning for ['InputTargetDataTransformer_ETSForecaster']
[2024-08-15 19:16:37,804] [automlx.dataset] Forecast horizon set to 13 for validation sets.
[2024-08-15 19:16:53,762] [automlx.trials] Best parameters for InputTargetDataTransformer_ETSForecaster: {'error': 'add', 'trend': 'mul', 'damped_trend': True, 'seasonal': 'add', 'seasonal_periods': 52, 'initialization_method': 'estimated', 'initial_level': None, 'initial_trend': None, 'initial_seasonal': None, 'bounds': None, 'dates': None, 'freq': None, 'missing': 'none', 'start_params': None, 'maxiter': 1500, 'disp': -1, 'return_params': False}
[2024-08-15 19:16:53,764] [automlx.trials] Model Tuning completed. Took: 16.138 secs
[2024-08-15 19:16:54,267] [automlx.trials] Running Model Tuning for ['InputTargetDataTransformer_STLwARIMAForecaster']
[2024-08-15 19:16:54,474] [automlx.dataset] Forecast horizon set to 13 for validation sets.
[2024-08-15 19:16:59,525] [automlx.trials] Best parameters for InputTargetDataTransformer_STLwARIMAForecaster: {'seasonal_deg': 0, 'trend_deg': 1, 'low_pass_deg': 0, 'period': 52, 'arima_p': 2, 'arima_d': 1, 'arima_q': 2, 'arima_trend': 'n', 'concentrate_scale': True}
[2024-08-15 19:16:59,527] [automlx.trials] Model Tuning completed. Took: 5.259 secs
[2024-08-15 19:17:03,924] [automlx.interface] Re-fitting pipeline
[2024-08-15 19:17:08,635] [automlx.interface] AutoMLx completed.
Selected model: ETSForecaster
Selected model params: {'error': 'add', 'trend': 'mul', 'damped_trend': True, 'seasonal': 'add', 'seasonal_periods': 52, 'initialization_method': 'estimated', 'initial_level': None, 'initial_trend': None, 'initial_seasonal': None, 'bounds': None, 'dates': None, 'freq': None, 'missing': 'none', 'start_params': None, 'maxiter': 1500, 'disp': -1, 'return_params': False}

Generating and visualizing forecasts¶

There are two interfaces that support generating future forecasts using the trained forecasting pipeline. The preferred function, forecast(), accepts a user-input value for the number of periods to forecast into the future, i.e., relative to the end of the training series. It also accepts a significance level to generate prediction confidence intervals (CIs). When the methods support intervals, confidence intervals at 1-alpha are generated, for example, significance level alpha=0.05 generates 95% confidence intervals.

In [12]:
predictions = est1.forecast(periods=len(y_test), alpha=0.05)

The predict(X) interface supports absolute index-based forecasts, but does not support confidence intervals (CIs). It also downcasts the index to int64.

It should be utilized only when you want to get both in-sample predictions (predictions at timestamps that are part of the training set) and out-of-sample predictions (predictions at timestamps that are not in the training set). The X variable should be an empty dataframe containing only the requested timestamps as index. Here we request 5 in-sample model fit values and 5 out-of-sample forecasts.

In [13]:
future_index = y_train.index[-5:].union(y_test.index[:5])
print(future_index)
DatetimeIndex(['2016-03-06 12:00:00', '2016-03-13 12:00:00',
               '2016-03-20 12:00:00', '2016-03-27 12:00:00',
               '2016-04-03 12:00:00', '2016-04-10 12:00:00',
               '2016-04-17 12:00:00', '2016-04-24 12:00:00',
               '2016-05-01 12:00:00', '2016-05-08 12:00:00'],
              dtype='datetime64[ns]', freq='W-SUN')
In [14]:
est1.predict(X=pd.DataFrame(index=future_index))
Out[14]:
Finance_W142
2016-03-06 12:00:00 5015.627932
2016-03-13 12:00:00 5013.775014
2016-03-20 12:00:00 4998.443940
2016-03-27 12:00:00 5013.689868
2016-04-03 12:00:00 5028.477761
2016-04-10 12:00:00 5052.891415
2016-04-17 12:00:00 5066.990374
2016-04-24 12:00:00 5080.142527
2016-05-01 12:00:00 5104.394553
2016-05-08 12:00:00 5090.115109

AutoML provides a simple one-line tool to visualize forecasts and confidence intervals.

In [15]:
est1.plot_forecast(predictions=predictions, additional_frames=dict(y_test=y_test))

Analyze the AutoML optimization process¶

During the AutoML process, a summary of the optimization process is logged. It consists of:

  • Information about the training data
  • Information about the AutoML Pipeline, such as:
    • selected features that AutoML found to be most predictive in the training data;
    • selected algorithm that was the best choice for this data;
    • hyperparameters for the selected algorithm.

AutoML provides a print_summary() API to output all the different trials performed.

In [16]:
est1.print_summary()

General Summary
Training Dataset size (174, 2)
Validation Dataset size None
CV TimeSeriesCV(Shuffle=False, Seed=7)
Optimization Metric neg_sym_mean_abs_percent_error
Selected Algorithm ETSForecaster
Selected Hyperparameters {'error': 'add', 'trend': 'mul', 'damped_trend': True, 'seasonal': 'add', 'seasonal_periods': 52, 'initialization_method': 'estimated', 'initial_level': None, 'initial_trend': None, 'initial_seasonal': None, 'bounds': None, 'dates': None, 'freq': None, 'missing': 'none', 'start_params': None, 'maxiter': 1500, 'disp': -1, 'return_params': False}
AutoMLx version 24.3.0
Python version 3.11.9 (main, Jul 18 2024, 13:17:53) [GCC 9.3.1 20200408 (Red Hat 9.3.1-2)]

Trials Summary
Step # Samples # Features Algorithm Hyperparameters Score (neg_sym_mean_abs_percent_error) Runtime (Seconds) Memory Usage (GB) Finished
0 Model Selection 161 2 InputTargetDataTransformer_ETSForecaster {'error': 'add', 'trend': 'add', 'damped_trend': False, 'seasonal': 'add', 'seasonal_periods': 52, 'initialization_method': 'estimated', 'initial_level': None, 'initial_trend': None, 'initial_seasonal': None, 'bounds': None, 'dates': None, 'freq': None, 'missing': 'none', 'start_params': None, 'maxiter': 1500, 'disp': -1, 'return_params': False} -0.036 21.8211 0.2593 Thu Aug 15 19:16:32 2024
1 Model Selection 161 2 InputTargetDataTransformer_STLwARIMAForecaster {'seasonal_deg': 1, 'trend_deg': 1, 'low_pass_deg': 0, 'period': 52, 'arima_p': 2, 'arima_d': 1, 'arima_q': 2, 'arima_trend': 'n', 'concentrate_scale': True} -0.0855 3.0864 0.3030 Thu Aug 15 19:16:29 2024
2 Model Selection 161 2 InputTargetDataTransformer_XGBForecaster {'differencing_order': 1, 'acf_local_maxima': '[52, 35, 17, 26, 22, 4, 48, 30, 13, 56, 39, 61, 44, 9, 65, 1]', 'use_X': True, 'n_estimators': 50, 'max_depth': 5} -0.0951 20.1390 0.7001 Thu Aug 15 19:16:34 2024
3 Model Selection 161 2 InputTargetDataTransformer_ExtraTreesForecaster {'differencing_order': 1, 'acf_local_maxima': '[52, 35, 17, 26, 22, 4, 48, 30, 13, 56, 39, 61, 44, 9, 65, 1]', 'use_X': True, 'n_estimators': 40, 'min_samples_leaf': 0.030458} -0.1252 20.1737 0.6971 Thu Aug 15 19:16:34 2024
4 Model Selection 161 2 InputTargetDataTransformer_LGBMForecaster {'differencing_order': 1, 'acf_local_maxima': '[52, 35, 17, 26, 22, 4, 48, 30, 13, 56, 39, 61, 44, 9, 65, 1]', 'use_X': True, 'max_depth': 4, 'n_estimators': 37} -0.13 18.2716 0.7028 Thu Aug 15 19:16:36 2024
5 Model Selection 122 2 InputTargetDataTransformer_ThetaForecaster {'sp': 52, 'deseasonalize': False, 'initial_level': None} -0.2091 7.0557 0.2933 Thu Aug 15 19:16:29 2024
6 Model Selection 148 2 InputTargetDataTransformer_ExpSmoothForecaster {'trend': 'add', 'damped_trend': True, 'seasonal': None, 'sp': 52, 'use_boxcox': False} -0.2475 6.5339 0.2853 Thu Aug 15 19:16:29 2024
7 Model Selection 109 2 InputTargetDataTransformer_NaiveForecaster {'strategy': 'last', 'sp': 52, 'window_length': None} -0.4494 6.6660 0.2788 Thu Aug 15 19:16:28 2024
8 Model Selection 161 2 InputTargetDataTransformer_STLwESForecaster {'seasonal_deg': 1, 'trend_deg': 1, 'low_pass_deg': 1, 'period': 52, 'es_trend': 'add', 'es_damped_trend': True, 'concentrate_scale': True} -inf 9.0492 0.2923 Thu Aug 15 19:16:30 2024
9 Model Selection 161 2 InputTargetDataTransformer_SARIMAXForecaster {'sp': 52, 'p': 2, 'd': 1, 'q': 2, 'P': 1, 'D': 0, 'Q': 1, 'trend': 'n', 'use_X': True, 'enforce_stationarity': True, 'enforce_invertibility': True, 'method': 'lbfgs', 'disp': -1, 'concentrate_scale': False} -inf 32.5628 0.3386 Thu Aug 15 19:16:35 2024
10 Model Tuning 161 2 InputTargetDataTransformer_ETSForecaster {'error': 'add', 'trend': 'mul', 'damped_trend': True, 'seasonal': 'add', 'seasonal_periods': 52, 'initialization_method': 'estimated', 'initial_level': None, 'initial_trend': None, 'initial_seasonal': None, 'bounds': None, 'dates': None, 'freq': None, 'missing': 'none', 'start_params': None, 'maxiter': 1500, 'disp': -1, 'return_params': False} -0.0327 19.0839 0.7035 Thu Aug 15 19:16:51 2024
11 Model Tuning 161 2 InputTargetDataTransformer_ETSForecaster {'error': 'add', 'trend': 'add', 'damped_trend': True, 'seasonal': 'add', 'seasonal_periods': 52, 'initialization_method': 'estimated', 'initial_level': None, 'initial_trend': None, 'initial_seasonal': None, 'bounds': None, 'dates': None, 'freq': None, 'missing': 'none', 'start_params': None, 'maxiter': 1500, 'disp': -1, 'return_params': False} -0.0334 14.7964 0.7031 Thu Aug 15 19:16:40 2024
12 Model Tuning 161 2 InputTargetDataTransformer_ETSForecaster {'error': 'mul', 'trend': 'add', 'damped_trend': True, 'seasonal': 'add', 'seasonal_periods': 52, 'initialization_method': 'estimated', 'initial_level': None, 'initial_trend': None, 'initial_seasonal': None, 'bounds': None, 'dates': None, 'freq': None, 'missing': 'none', 'start_params': None, 'maxiter': 1500, 'disp': -1, 'return_params': False} -0.036 17.6693 0.7036 Thu Aug 15 19:16:49 2024
13 Model Tuning 161 2 InputTargetDataTransformer_ETSForecaster {'error': 'add', 'trend': 'add', 'damped_trend': False, 'seasonal': 'add', 'seasonal_periods': 52, 'initialization_method': 'estimated', 'initial_level': None, 'initial_trend': None, 'initial_seasonal': None, 'bounds': None, 'dates': None, 'freq': None, 'missing': 'none', 'start_params': None, 'maxiter': 1500, 'disp': -1, 'return_params': False} -0.0361 13.3357 0.7024 Thu Aug 15 19:16:40 2024
14 Model Tuning 135 2 InputTargetDataTransformer_ETSForecaster {'error': 'add', 'trend': 'add', 'damped_trend': False, 'seasonal': 'add', 'seasonal_periods': 52, 'initialization_method': 'estimated', 'initial_level': None, 'initial_trend': None, 'initial_seasonal': None, 'bounds': None, 'dates': None, 'freq': None, 'missing': 'none', 'start_params': None, 'maxiter': 1500, 'disp': -1, 'return_params': False} -0.0361 12.6781 0.7034 Thu Aug 15 19:16:43 2024
15 Model Tuning 148 2 InputTargetDataTransformer_ETSForecaster {'error': 'add', 'trend': 'add', 'damped_trend': False, 'seasonal': 'add', 'seasonal_periods': 52, 'initialization_method': 'estimated', 'initial_level': None, 'initial_trend': None, 'initial_seasonal': None, 'bounds': None, 'dates': None, 'freq': None, 'missing': 'none', 'start_params': None, 'maxiter': 1500, 'disp': -1, 'return_params': False} -0.0361 13.1573 0.7020 Thu Aug 15 19:16:44 2024
16 Model Tuning 161 2 InputTargetDataTransformer_ETSForecaster {'error': 'add', 'trend': None, 'damped_trend': True, 'seasonal': 'add', 'seasonal_periods': 52, 'initialization_method': 'estimated', 'initial_level': None, 'initial_trend': None, 'initial_seasonal': None, 'bounds': None, 'dates': None, 'freq': None, 'missing': 'none', 'start_params': None, 'maxiter': 1500, 'disp': -1, 'return_params': False} -0.0469 12.3479 0.7044 Thu Aug 15 19:16:53 2024
17 Model Tuning 161 2 InputTargetDataTransformer_ETSForecaster {'error': 'add', 'trend': None, 'damped_trend': False, 'seasonal': 'add', 'seasonal_periods': 52, 'initialization_method': 'estimated', 'initial_level': None, 'initial_trend': None, 'initial_seasonal': None, 'bounds': None, 'dates': None, 'freq': None, 'missing': 'none', 'start_params': None, 'maxiter': 1500, 'disp': -1, 'return_params': False} -0.0469 13.3312 0.7043 Thu Aug 15 19:16:47 2024
18 Model Tuning 161 2 InputTargetDataTransformer_STLwARIMAForecaster {'seasonal_deg': 0, 'trend_deg': 1, 'low_pass_deg': 0, 'period': 52, 'arima_p': 2, 'arima_d': 1, 'arima_q': 2, 'arima_trend': 'n', 'concentrate_scale': True} -0.0551 2.8733 0.7115 Thu Aug 15 19:16:57 2024
19 Model Tuning 161 2 InputTargetDataTransformer_STLwARIMAForecaster {'seasonal_deg': 1, 'trend_deg': 1, 'low_pass_deg': 1, 'period': 52, 'arima_p': 2, 'arima_d': 1, 'arima_q': 2, 'arima_trend': 'n', 'concentrate_scale': True} -0.0839 2.3810 0.7116 Thu Aug 15 19:16:56 2024
20 Model Tuning 161 2 InputTargetDataTransformer_STLwARIMAForecaster {'seasonal_deg': 1, 'trend_deg': 1, 'low_pass_deg': 0, 'period': 52, 'arima_p': 2, 'arima_d': 1, 'arima_q': 2, 'arima_trend': 'n', 'concentrate_scale': True} -0.0855 2.6034 0.7059 Thu Aug 15 19:16:56 2024
21 Model Tuning 135 2 InputTargetDataTransformer_STLwARIMAForecaster {'seasonal_deg': 1, 'trend_deg': 1, 'low_pass_deg': 0, 'period': 52, 'arima_p': 2, 'arima_d': 1, 'arima_q': 2, 'arima_trend': 'n', 'concentrate_scale': True} -0.0855 2.5809 0.7120 Thu Aug 15 19:16:56 2024
22 Model Tuning 161 2 InputTargetDataTransformer_STLwARIMAForecaster {'seasonal_deg': 1, 'trend_deg': 1, 'low_pass_deg': 0, 'period': 52, 'arima_p': 2, 'arima_d': 1, 'arima_q': 2, 'arima_trend': 'n', 'concentrate_scale': True} -0.0855 2.2540 0.7117 Thu Aug 15 19:16:57 2024
23 Model Tuning 161 2 InputTargetDataTransformer_STLwARIMAForecaster {'seasonal_deg': 1, 'trend_deg': 1, 'low_pass_deg': 0, 'period': 52, 'arima_p': 2, 'arima_d': 1, 'arima_q': 2, 'arima_trend': 'n', 'concentrate_scale': True} -0.0855 2.3840 0.7057 Thu Aug 15 19:16:57 2024
24 Model Tuning 161 2 InputTargetDataTransformer_STLwARIMAForecaster {'seasonal_deg': 1, 'trend_deg': 1, 'low_pass_deg': 0, 'period': 52, 'arima_p': 2, 'arima_d': 1, 'arima_q': 2, 'arima_trend': 'n', 'concentrate_scale': True} -0.0855 2.4665 0.7135 Thu Aug 15 19:16:56 2024
25 Model Tuning 161 2 InputTargetDataTransformer_STLwARIMAForecaster {'seasonal_deg': 1, 'trend_deg': 1, 'low_pass_deg': 0, 'period': 52, 'arima_p': 2, 'arima_d': 1, 'arima_q': 2, 'arima_trend': 'n', 'concentrate_scale': True} -0.0855 2.9517 0.7114 Thu Aug 15 19:16:55 2024
26 Model Tuning 161 2 InputTargetDataTransformer_STLwARIMAForecaster {'seasonal_deg': 1, 'trend_deg': 1, 'low_pass_deg': 0, 'period': 52, 'arima_p': 1, 'arima_d': 1, 'arima_q': 2, 'arima_trend': 'n', 'concentrate_scale': True} -0.0891 2.6177 0.7112 Thu Aug 15 19:16:55 2024
27 Model Tuning 161 2 InputTargetDataTransformer_STLwARIMAForecaster {'seasonal_deg': 1, 'trend_deg': 1, 'low_pass_deg': 0, 'period': 52, 'arima_p': 2, 'arima_d': 1, 'arima_q': 1, 'arima_trend': 'n', 'concentrate_scale': True} -0.0962 2.0384 0.7111 Thu Aug 15 19:16:56 2024
28 Model Tuning 161 2 InputTargetDataTransformer_STLwARIMAForecaster {'seasonal_deg': 1, 'trend_deg': 1, 'low_pass_deg': 0, 'period': 52, 'arima_p': 2, 'arima_d': 1, 'arima_q': 0, 'arima_trend': 'n', 'concentrate_scale': True} -0.1188 1.6989 0.7112 Thu Aug 15 19:16:55 2024
29 Model Tuning 161 2 InputTargetDataTransformer_STLwARIMAForecaster {'seasonal_deg': 1, 'trend_deg': 1, 'low_pass_deg': 0, 'period': 52, 'arima_p': 0, 'arima_d': 1, 'arima_q': 2, 'arima_trend': 'n', 'concentrate_scale': True} -0.1198 2.0501 0.7121 Thu Aug 15 19:16:54 2024
30 Model Tuning 161 2 InputTargetDataTransformer_ETSForecaster {'error': 'add', 'trend': 'add', 'damped_trend': False, 'seasonal': 'mul', 'seasonal_periods': 52, 'initialization_method': 'estimated', 'initial_level': None, 'initial_trend': None, 'initial_seasonal': None, 'bounds': None, 'dates': None, 'freq': None, 'missing': 'none', 'start_params': None, 'maxiter': 1500, 'disp': -1, 'return_params': False} -0.1244 15.2929 0.7039 Thu Aug 15 19:16:44 2024
31 Model Tuning 161 2 InputTargetDataTransformer_STLwARIMAForecaster {'seasonal_deg': 0, 'trend_deg': 0, 'low_pass_deg': 0, 'period': 52, 'arima_p': 5, 'arima_d': 1, 'arima_q': 0, 'arima_trend': 'n', 'concentrate_scale': True} -0.1274 2.2509 0.7121 Thu Aug 15 19:16:58 2024
32 Model Tuning 161 2 InputTargetDataTransformer_ETSForecaster {'error': 'add', 'trend': 'add', 'damped_trend': True, 'seasonal': 'mul', 'seasonal_periods': 52, 'initialization_method': 'estimated', 'initial_level': None, 'initial_trend': None, 'initial_seasonal': None, 'bounds': None, 'dates': None, 'freq': None, 'missing': 'none', 'start_params': None, 'maxiter': 1500, 'disp': -1, 'return_params': False} -0.129 14.4294 0.7005 Thu Aug 15 19:16:49 2024
33 Model Tuning 161 2 InputTargetDataTransformer_STLwARIMAForecaster {'seasonal_deg': 0, 'trend_deg': 0, 'low_pass_deg': 0, 'period': 52, 'arima_p': 1, 'arima_d': 1, 'arima_q': 0, 'arima_trend': 'n', 'concentrate_scale': True} -0.135 1.6667 0.7119 Thu Aug 15 19:16:58 2024
34 Model Tuning 161 2 InputTargetDataTransformer_STLwARIMAForecaster {'seasonal_deg': 0, 'trend_deg': 0, 'low_pass_deg': 0, 'period': 52, 'arima_p': 2, 'arima_d': 1, 'arima_q': 0, 'arima_trend': 'n', 'concentrate_scale': True} -0.1359 1.8356 0.7061 Thu Aug 15 19:16:58 2024
35 Model Tuning 161 2 InputTargetDataTransformer_STLwARIMAForecaster {'seasonal_deg': 0, 'trend_deg': 0, 'low_pass_deg': 0, 'period': 52, 'arima_p': 3, 'arima_d': 1, 'arima_q': 0, 'arima_trend': 'n', 'concentrate_scale': True} -0.1376 2.4579 0.7141 Thu Aug 15 19:16:58 2024
36 Model Tuning 161 2 InputTargetDataTransformer_ETSForecaster {'error': 'add', 'trend': 'mul', 'damped_trend': False, 'seasonal': 'add', 'seasonal_periods': 52, 'initialization_method': 'estimated', 'initial_level': None, 'initial_trend': None, 'initial_seasonal': None, 'bounds': None, 'dates': None, 'freq': None, 'missing': 'none', 'start_params': None, 'maxiter': 1500, 'disp': -1, 'return_params': False} -0.1409 15.7406 0.7036 Thu Aug 15 19:16:47 2024
37 Model Tuning 161 2 InputTargetDataTransformer_ETSForecaster {'error': 'add', 'trend': 'add', 'damped_trend': True, 'seasonal': 'add', 'seasonal_periods': 1, 'initialization_method': 'estimated', 'initial_level': None, 'initial_trend': None, 'initial_seasonal': None, 'bounds': None, 'dates': None, 'freq': None, 'missing': 'none', 'start_params': None, 'maxiter': 1500, 'disp': -1, 'return_params': False} -0.2209 1.4717 0.7033 Thu Aug 15 19:16:48 2024
38 Model Tuning 161 2 InputTargetDataTransformer_ETSForecaster {'error': 'add', 'trend': 'add', 'damped_trend': True, 'seasonal': None, 'seasonal_periods': 52, 'initialization_method': 'estimated', 'initial_level': None, 'initial_trend': None, 'initial_seasonal': None, 'bounds': None, 'dates': None, 'freq': None, 'missing': 'none', 'start_params': None, 'maxiter': 1500, 'disp': -1, 'return_params': False} -0.2209 1.4841 0.7041 Thu Aug 15 19:16:47 2024
39 Model Tuning 161 2 InputTargetDataTransformer_STLwARIMAForecaster {'seasonal_deg': 1, 'trend_deg': 1, 'low_pass_deg': 0, 'period': 35, 'arima_p': 2, 'arima_d': 1, 'arima_q': 2, 'arima_trend': 'n', 'concentrate_scale': True} -0.2295 2.0397 0.7116 Thu Aug 15 19:16:56 2024
40 Model Tuning 161 2 InputTargetDataTransformer_STLwARIMAForecaster {'seasonal_deg': 1, 'trend_deg': 0, 'low_pass_deg': 0, 'period': 52, 'arima_p': 2, 'arima_d': 1, 'arima_q': 2, 'arima_trend': 'n', 'concentrate_scale': True} -0.2544 2.4226 0.7123 Thu Aug 15 19:16:57 2024
41 Model Tuning 161 2 InputTargetDataTransformer_ETSForecaster {'error': 'add', 'trend': 'add', 'damped_trend': False, 'seasonal': 'add', 'seasonal_periods': 1, 'initialization_method': 'estimated', 'initial_level': None, 'initial_trend': None, 'initial_seasonal': None, 'bounds': None, 'dates': None, 'freq': None, 'missing': 'none', 'start_params': None, 'maxiter': 1500, 'disp': -1, 'return_params': False} -0.2919 1.5223 0.7032 Thu Aug 15 19:16:43 2024
42 Model Tuning 161 2 InputTargetDataTransformer_ETSForecaster {'error': 'add', 'trend': 'add', 'damped_trend': False, 'seasonal': None, 'seasonal_periods': 52, 'initialization_method': 'estimated', 'initial_level': None, 'initial_trend': None, 'initial_seasonal': None, 'bounds': None, 'dates': None, 'freq': None, 'missing': 'none', 'start_params': None, 'maxiter': 1500, 'disp': -1, 'return_params': False} -0.2919 1.4820 0.7017 Thu Aug 15 19:16:41 2024
43 Model Tuning 161 2 InputTargetDataTransformer_ETSForecaster {'error': 'add', 'trend': 'add', 'damped_trend': False, 'seasonal': 'add', 'seasonal_periods': 35, 'initialization_method': 'estimated', 'initial_level': None, 'initial_trend': None, 'initial_seasonal': None, 'bounds': None, 'dates': None, 'freq': None, 'missing': 'none', 'start_params': None, 'maxiter': 1500, 'disp': -1, 'return_params': False} -inf 14.4222 0.7011 Thu Aug 15 19:16:45 2024
44 Model Tuning 161 2 InputTargetDataTransformer_ETSForecaster {'error': 'add', 'trend': 'add', 'damped_trend': False, 'seasonal': 'add', 'seasonal_periods': 52, 'initialization_method': 'estimated', 'initial_level': None, 'initial_trend': None, 'initial_seasonal': None, 'bounds': None, 'dates': None, 'freq': None, 'missing': 'none', 'start_params': None, 'maxiter': 1500, 'disp': -1, 'return_params': False} -inf 14.5241 0.7018 Thu Aug 15 19:16:47 2024
45 Model Tuning 161 2 InputTargetDataTransformer_ETSForecaster {'error': 'add', 'trend': 'add', 'damped_trend': False, 'seasonal': 'add', 'seasonal_periods': 52, 'initialization_method': 'estimated', 'initial_level': None, 'initial_trend': None, 'initial_seasonal': None, 'bounds': None, 'dates': None, 'freq': None, 'missing': 'none', 'start_params': None, 'maxiter': 1500, 'disp': -1, 'return_params': False} -inf 16.0069 0.7010 Thu Aug 15 19:16:41 2024
46 Model Tuning 161 2 InputTargetDataTransformer_ETSForecaster {'error': 'add', 'trend': 'add', 'damped_trend': True, 'seasonal': 'add', 'seasonal_periods': 35, 'initialization_method': 'estimated', 'initial_level': None, 'initial_trend': None, 'initial_seasonal': None, 'bounds': None, 'dates': None, 'freq': None, 'missing': 'none', 'start_params': None, 'maxiter': 1500, 'disp': -1, 'return_params': False} -inf 14.5421 0.7042 Thu Aug 15 19:16:50 2024
47 Model Tuning 161 2 InputTargetDataTransformer_ETSForecaster {'error': 'mul', 'trend': 'add', 'damped_trend': False, 'seasonal': 'add', 'seasonal_periods': 52, 'initialization_method': 'estimated', 'initial_level': None, 'initial_trend': None, 'initial_seasonal': None, 'bounds': None, 'dates': None, 'freq': None, 'missing': 'none', 'start_params': None, 'maxiter': 1500, 'disp': -1, 'return_params': False} -inf 22.2689 0.7002 Thu Aug 15 19:16:42 2024
48 Model Tuning 161 2 InputTargetDataTransformer_STLwARIMAForecaster {'seasonal_deg': 0, 'trend_deg': 0, 'low_pass_deg': 0, 'period': 52, 'arima_p': 0, 'arima_d': 1, 'arima_q': 0, 'arima_trend': 'n', 'concentrate_scale': True} -inf 1.4146 0.7141 Thu Aug 15 19:16:57 2024
49 Model Tuning 161 2 InputTargetDataTransformer_STLwARIMAForecaster {'seasonal_deg': 0, 'trend_deg': 0, 'low_pass_deg': 0, 'period': 52, 'arima_p': 0, 'arima_d': 1, 'arima_q': 0, 'arima_trend': 'n', 'concentrate_scale': True} -inf 1.4281 0.7120 Thu Aug 15 19:16:59 2024
50 Model Tuning 161 2 InputTargetDataTransformer_STLwARIMAForecaster {'seasonal_deg': 1, 'trend_deg': 1, 'low_pass_deg': 0, 'period': 1, 'arima_p': 2, 'arima_d': 1, 'arima_q': 2, 'arima_trend': 'n', 'concentrate_scale': True} -inf 1.5674 0.7139 Thu Aug 15 19:16:56 2024
51 Model Tuning 161 2 InputTargetDataTransformer_STLwARIMAForecaster {'seasonal_deg': 1, 'trend_deg': 1, 'low_pass_deg': 0, 'period': 52, 'arima_p': 3, 'arima_d': 1, 'arima_q': 2, 'arima_trend': 'n', 'concentrate_scale': True} -inf 3.3569 0.7005 Thu Aug 15 19:16:55 2024
52 Model Tuning 122 2 InputTargetDataTransformer_STLwARIMAForecaster {'seasonal_deg': 1, 'trend_deg': 1, 'low_pass_deg': 0, 'period': 52, 'arima_p': 5, 'arima_d': 1, 'arima_q': 2, 'arima_trend': 'n', 'concentrate_scale': True} -inf 3.7713 0.7135 Thu Aug 15 19:16:56 2024

We also provide the capability to visualize the results of each stage of the AutoML pipeline.

Algorithm Selection¶

The plot below shows the scores predicted by Algorithm Selection for each algorithm. Since negative sMAPE is used by default, higher values (closer to zero) are better. The horizontal line shows the average score across all algorithms. Algorithms with better score than average are colored turquoise, whereas those with worse score than average are colored teal.

In [17]:
# Each trial is a tuple of
# (algorithm, no. samples, no. features, mean CV score, hyperparameters,
# all CV scores, total CV time (s), memory usage (Gb))
trials = est1.completed_trials_summary_[est1.completed_trials_summary_["Step"].str.contains('Model Selection')]
name_of_score_column = f"Score ({est1._inferred_score_metric[0].name})"
trials.replace([np.inf, -np.inf], np.nan, inplace=True)
trials.dropna(subset=[name_of_score_column], inplace = True)
scores = trials[name_of_score_column].tolist()
models = trials['Algorithm'].tolist()
colors = []

y_margin = 0.10 * (max(scores) - min(scores))
s = pd.Series(scores, index=models).sort_values(ascending=False)
s = s.dropna()
for f in s.keys():
    if f.strip()  ==  est1.selected_model_.strip():
        colors.append('orange')
    elif s[f] >= s.mean():
        colors.append('teal')
    else:
        colors.append('turquoise')



fig, ax = plt.subplots(1)
ax.set_title("Algorithm Selection Trials")
ax.set_ylim(min(scores) - y_margin, max(scores) + y_margin)
ax.set_ylabel(est1._inferred_score_metric[0].name)
s.plot.bar(ax=ax, color=colors, edgecolor='black')
ax.axhline(y=s.mean(), color='black', linewidth=0.5)
plt.show()

Hyperparameter Tuning¶

Hyperparameter tuning is the last stage of the Oracle AutoMLx pipeline, and focuses on improving the chosen algorithm's score. We use a novel algorithm to search across many hyperparameter dimensions, and converge automatically when optimal hyperparameters are identified. Each trial in the graph below represents a particular hyperparameter combination for the selected model.

In [18]:
# Each trial is a row in a dataframe that contains
# Algorithm, Number of Samples, Number of Features, Hyperparameters, Score, Runtime, Memory Usage, Step as features
trials = est1.completed_trials_summary_[est1.completed_trials_summary_["Step"].str.contains('Model Tuning')]
trials.replace([np.inf, -np.inf], np.nan, inplace=True)
trials.dropna(subset=[name_of_score_column], inplace = True)
trials.drop(trials[trials['Finished'] == -1].index, inplace = True)
trials['Finished']= trials['Finished'].apply(lambda x: time.mktime(datetime.datetime.strptime(x,
                                             "%a %b %d %H:%M:%S %Y").timetuple()))
trials.sort_values(by=['Finished'],ascending=True, inplace = True)
scores = trials[name_of_score_column].tolist()
score = []
score.append(scores[0])
for i in range(1,len(scores)):
    if scores[i]>= score[i-1]:
        score.append(scores[i])
    else:
        score.append(score[i-1])
y_margin = 0.10 * (max(score) - min(score))

fig, ax = plt.subplots(1)
ax.set_title("Hyperparameter Tuning Trials")
ax.set_xlabel("Iteration $n$")
ax.set_ylabel(est1._inferred_score_metric[0].name)
ax.grid(color='g', linestyle='-', linewidth=0.1)
ax.set_ylim(min(score) - y_margin, max(score) + y_margin)
ax.plot(range(1, len(trials) + 1), score, 'k:', marker="s", color='teal', markersize=3)
plt.show()

We can also view all tuned algorithms, as well as their validation and testing performance. This provides a good sanity check for the decision making.

In [19]:
print(f'### Plotting is enabled for total models tuned = {len(est1.pipelines_)}.')
print('### Model_name\t\t Val_score ')
model_name = ""
if len(est1.pipelines_) > 1:
    model_name = "InputTargetDataTransformer_"

for pipeline in est1.pipelines_:
    print(  pipeline.selected_model_, " \t",
            '%.4f'%est1.completed_trials_summary_[est1.completed_trials_summary_["Algorithm"] == model_name+pipeline.selected_model_].iloc[0][name_of_score_column],'\t',   # validation_score
         )
    predictions = pipeline.forecast(periods=len(y_test), alpha=0.05)                     # out-of-sample forecast
    predictions.index = y_test.index
    fig = pipeline.plot_forecast(predictions=predictions, additional_frames=dict(y_test=y_test))
    fig.show()
### Plotting is enabled for total models tuned = 2.
### Model_name		 Val_score 
ETSForecaster  	 -0.0360 	
STLwARIMAForecaster  	 -0.0855 	

Multivariate time series¶

We now display the use of the Oracle AutoMLx solution for multivariate time series.

There are two types of multivariate time series in forecasting:

  • Single-target forecasting, with additional variables
  • Multi-target forecasting, with or without additional variables

A time series target is the variable that the forecaster will try to predict. Additional variables provide more information that help with the prediction of the target. In single-target forecasting, we are forecasting a single variable, whereas in multi-target forecasting, we forecast values of highly correlated variables together as a system.

We will first see an example of single-target forecasting with additional variables and then a second example with a multi-target forecasting problem.

Single-target Forecasting with Exogenous Variables¶

We will now take two time series from the M4 dataset: one target variable, and one additional variable used solely as explanatory variables. Note that $X$ consists of the additional variables ('W142') while $y$ has the target variables ('W143'). We will also showcase AutoML's functionality in the absence of a datetime index, by dropping the datetime index and utilize the series with only an int64index.

In [20]:
series_ids = ["W142", "W143"]
series_values = all_series.loc[series_ids].T

# drop NaNs for the time period where data wasn't recorded
series_values.dropna(inplace=True)

# we will take the first 100 elements of the series
series_values = series_values[:100]

# retrieve starting date of recording and series length to generate the datetime index
start_date = "28-06-00 12:00"
future_dates = pd.date_range(start=start_date, periods=len(series_values), freq="W")
data = pd.DataFrame(series_values.to_numpy(), index=future_dates, columns=series_ids)
data.index = np.arange(0, len(data))

y = pd.DataFrame(data["W142"])
X = pd.DataFrame(data["W143"])

We then split it using a temporal train-test split as done previously.

In [21]:
X_train_df, X_test_df = temporal_train_test_split(X, train_size=0.9)
y_train_df, y_test_df = temporal_train_test_split(y, train_size=0.9)

We will now fit the pipeline on this data. For the single-target forecasting task with additional variables, the pipeline considers all available models by default. However, the models that actually support additional variables are the following:

  • LGBMForecaster
  • ExtraTreesForecaster
  • XGBForecaster
  • SARIMAXForecaster
  • ProphetForecaster These models also have the option to discard additional variables. This can be set as a boolean hyperparameter called use_X.
In [22]:
est2 = automlx.Pipeline(task='forecasting',
                        model_list=['LGBMForecaster', 
                                    'ExtraTreesForecaster',  
                                    'XGBForecaster', 
                                    'SARIMAXForecaster', 
                                    'ProphetForecaster'])
est2.fit(X=X_train_df, y=y_train_df)

print('Selected model: {}'.format(est2.selected_model_))
print('Selected model params: {}'.format(est2.selected_model_params_))
[2024-08-15 19:17:12,297] [automlx.data_transform] Number of simple differencing orders required: d = 0
[2024-08-15 19:17:12,300] [automlx.data_transform] Seasonal Periodicities; from decomposed/adjusted: [1]
[2024-08-15 19:17:12,549] [automlx.interface] Dataset shape: (90,3)
[2024-08-15 19:17:12,877] [automlx.data_transform] Running preprocessing. Number of features: 4
[2024-08-15 19:17:13,134] [automlx.data_transform] Preprocessing completed. Took 0.257 secs
[2024-08-15 19:17:13,439] [automlx.process] Running Model Generation
[2024-08-15 19:17:13,520] [automlx] Provided model (SARIMAXForecaster) is not supported.Supported models are: ['LGBMForecaster', 'ExtraTreesForecaster', 'ExpSmoothForecaster', 'NaiveForecaster', 'ThetaForecaster', 'ETSForecaster', 'XGBForecaster']
[2024-08-15 19:17:13,521] [automlx] Provided model (ProphetForecaster) is not supported.Supported models are: ['LGBMForecaster', 'ExtraTreesForecaster', 'ExpSmoothForecaster', 'NaiveForecaster', 'ThetaForecaster', 'ETSForecaster', 'XGBForecaster']
[2024-08-15 19:17:13,523] [automlx.process] Model Generation completed.
[2024-08-15 19:17:13,689] [automlx.model_selection] Running Model Selection
[2024-08-15 19:17:21,086] [automlx.model_selection] Model Selection completed - Took 7.398 sec - Selected models: ['XGBForecaster']
[2024-08-15 19:17:21,567] [automlx.trials] Running Model Tuning for ['XGBForecaster']
[2024-08-15 19:17:36,683] [automlx.trials] Best parameters for XGBForecaster: {'differencing_order': 1, 'acf_local_maxima': '[4, 9, 18, 22, 14, 31, 26, 12, 1]', 'use_X': True, 'n_estimators': 16, 'max_depth': 2}
[2024-08-15 19:17:36,685] [automlx.trials] Model Tuning completed. Took: 15.118 secs
[2024-08-15 19:17:37,868] [automlx.interface] Re-fitting pipeline
[2024-08-15 19:17:41,159] [automlx.interface] AutoMLx completed.
Selected model: XGBForecaster
Selected model params: {'differencing_order': 1, 'acf_local_maxima': '[4, 9, 18, 22, 14, 31, 26, 12, 1]', 'use_X': True, 'n_estimators': 16, 'max_depth': 2}

Multi-target Forecasting with Exogenous Variables¶

For this use case, we now take 3 variables from the M4 dataset: two target variables (W142, W143) and one additional variables (W141). We will add a constant variable as an additional variable, so we have a total of two additional variables. We then split the data using a temporal train-test split and drop the datetime index, as done previously,

In [23]:
series_ids = ["W141", "W142", "W143"]
series_values = all_series.loc[series_ids].T

# drop NaNs for the time period where data wasn't recorded
series_values.dropna(inplace=True)
# we will take the first 100 elements of the series
series_values = series_values[:100]


# retrieve starting date of recording and series length to generate the datetime index
start_date = "28-06-00 12:00"
future_dates = pd.date_range(start=start_date, periods=len(series_values), freq="W")
data = pd.DataFrame(series_values.to_numpy(), index=future_dates, columns=series_ids)
data.index = np.arange(0, len(data))

y = data[["W142", "W143"]]
X = pd.DataFrame(data["W141"])
# adding the constant variable
X["const"] = 1

X_train_df, X_test_df = temporal_train_test_split(X, train_size=0.9)
y_train_df, y_test_df = temporal_train_test_split(y, train_size=0.9)

Advanced AutoML Configuration¶

You can also configure the pipeline with suitable parameters according to your needs. For the Multi-target forecasting task, the pipeline only considers two models : VARMAX and DynFactor.

In [24]:
custom_pipeline = automlx.Pipeline(
    task='forecasting',       # Supports 'classification', regression', 'anomaly_detection' and 'forecasting'
    model_list=[                 # Specify the models you want the AutoMLx to consider
        'DynFactorForecaster',
        'VARMAXForecaster'
    ],
    n_algos_tuned=2,             # Choose how many models to tune
    preprocessing=True,          # Disable or enable Preprocessing step. Default to `True`
    search_space={},             # You can specify the hyper-parameters and ranges we search
    max_tuning_trials=2,         # The maximum number of tuning trials. Can be integer or Dict (max number for each model)
    score_metric='neg_sym_mean_abs_percent_error',     # Any scikit-learn metric or a custom function
    time_series_period=None      # The period of time series
)

Specify the number of cross-validation (CV) folds¶

AutoML automatically decides how many folds to create, given the length of the input series. This is dependent on the frequency and length of the series. In the above, the preprocessor chose to create two folds. In the following we set the number of folds to 8.

In [25]:
custom_pipeline.fit(X=X_train_df, y=y_train_df, cv=8)
[2024-08-15 19:17:41,811] [automlx.data_transform] Number of simple differencing orders required: d = 0
[2024-08-15 19:17:41,815] [automlx.data_transform] Seasonal Periodicities; from decomposed/adjusted: [1]
[2024-08-15 19:17:42,153] [automlx.interface] Dataset shape: (90,5)
[2024-08-15 19:17:42,229] [automlx.interface] Model Tune disabled.
[2024-08-15 19:17:42,486] [automlx.data_transform] Running preprocessing. Number of features: 6
[2024-08-15 19:17:42,713] [automlx.data_transform] Preprocessing completed. Took 0.227 secs
[2024-08-15 19:17:43,029] [automlx.process] Running Model Generation
[2024-08-15 19:17:43,143] [automlx.process] Model Generation completed.
[2024-08-15 19:17:43,555] [automlx.model_selection] Running Model Selection
[2024-08-15 19:17:43,555] [automlx.trials] Fewer models (2) than top_k (2) provided, skipping model selection
[2024-08-15 19:17:43,644] [automlx.interface] Re-fitting pipeline
[2024-08-15 19:17:46,091] [automlx.interface] AutoMLx completed.
Out[25]:
<automlx._interface.forecaster.AutoForecaster at 0x7efcb15d9ed0>

The AutoML pipeline provides attributes to get the selected features, the chosen model, hyperparameters as well as the score on the test set.

In [26]:
test_score = custom_pipeline.score(X=X_test_df,y=y_test_df)

print('Ranked models: {}'.format(custom_pipeline.ranked_models_))
print('Selected model: {}'.format(custom_pipeline.selected_model_))
print('Selected model params: {}'.format(custom_pipeline.selected_model_params_))
print(f'Score on test data : {test_score}')
[2024-08-15 19:17:46,479] [automlx.model] Predict called on DynFactorForecaster that's not been fitted.
Ranked models: ['DynFactorForecaster', 'VARMAXForecaster']
Selected model: DynFactorForecaster
Selected model params: {'k_factors': 1, 'error_order': 2, 'factor_order': 2, 'error_cov_type': 'diagonal', 'use_X': True, 'error_var': False, 'enforce_stationarity': True}
Score on test data : -0.4736945883029529

Make predictions¶

As mention in the univariate data case, there are two ways of making a prediction :

  • forecast(k) allows one to predict k steps after the end of the training data. It should be used when one wants to make out-of-sample predictions
  • predict(X) returns predictions at the timestamps given as argument. It should be used when one wants to make in-sample predictions and out-of-sample predictions. It does not support confidence intervals.

In the cell below predict() is used on the last 5 timestamps of the train set, and all timestamps of the test set. forecast() is used to predict k steps after the training set, where k is the size of the test set.

In [27]:
y_pred = custom_pipeline.predict(pd.concat([X_train_df[-5:0],X_test_df], axis=0) )
y_forecast = custom_pipeline.forecast(len(y_test_df), alpha=0.8, X=X_test_df)    # out-of-sample forecast
[2024-08-15 19:17:46,597] [automlx.model] Predict called on DynFactorForecaster that's not been fitted.
[2024-08-15 19:17:46,639] [automlx.model] Forecast called on DynFactorForecaster that's not been fitted.

The obtained forecast contains predictions for the two target variables, as well as lower and upper confidence intervals, for each timestamp in the test set.

In [28]:
y_forecast
Out[28]:
W142 W143 W142_ci_lower W142_ci_upper
90 1523.315370 4120.632282 1514.232711 1532.398028
91 1529.220991 4110.701763 1515.520362 1542.921620
92 1516.883719 4118.868327 1499.729383 1534.038055
93 1512.790195 4119.127842 1492.815650 1532.764741
94 1512.239464 4115.999969 1489.846346 1534.632581
95 1504.476192 4119.899345 1479.947498 1529.004886
96 1504.039436 4116.732328 1477.588597 1530.490274
97 1498.492695 4118.553261 1470.288723 1526.696668
98 1502.087490 4111.546114 1472.268694 1531.906287
99 1493.464952 4116.418334 1462.147539 1524.782365

One can also directly compute the score of the tuned model on the test set, without needing to run forecast() or predict().

In [29]:
print("Tuned model testing score (negative sMAPE): ", custom_pipeline.score(X=X_test_df, y=y_test_df))
[2024-08-15 19:17:46,844] [automlx.model] Predict called on DynFactorForecaster that's not been fitted.
Tuned model testing score (negative sMAPE):  -0.4736945883029529

Visualization¶

Finally, when given as input the forecasted variables, the plot_forecast() method displays an interactive plot of the predictions (for each target variable) and confidence intervals.

In [30]:
custom_pipeline.plot_forecast(predictions=y_forecast, additional_frames=dict(test=y_test_df))

Machine Learning Explainability (MLX)¶

For a variety of decision-making tasks, getting only a prediction as model output is not sufficient. A user may wish to know why the model outputs that prediction, or which data features are relevant for that prediction. For that purpose the Oracle AutoMLx solution defines the MLExplainer object, which allows to compute multiple types of model explanations.

Initializing an MLExplainer¶

The MLExplainer object takes as arguments the trained model, the training data and labels, as well as the task. Let's assume that we want to learn more about the decision-making process of the model est1 that we trained based on the M4 dataset (Finance_W142) previously. Thus, we pass est1 along with its training series (y_train) to the MLExplainer to produce the appropriate explainer.

In [31]:
explainer = automlx.MLExplainer(est1,
                                None,
                                y_train,
                                task="forecasting")

Prediction Explanations (Comparative Feature Importance)¶

For any data point within the forecasting horizon, explanations indicate which factors increased or decreased the model's prediction compared to a previous reference point. This comparison provides additional context, making the information more interpretable for the user.

The function explain_prediction() calculates the comparative feature importance for a specified datapoint. For example, if you want to explore the model's predictions and understand the impact of various temporal concepts on the decision-making process for the forecast 10 time steps in the future, you can obtain the prediction explanation as shown below.

In [32]:
exp = explainer.explain_prediction(forecast_timepoints=[10])

Visualization¶

There are two options to show the explanation's results:

  • to_dataframe() will return a dataframe of the results. The concepts and features are returned in decreasing order of importance.
  • show_in_notebook() will present the results as a Waterfall plot, comparing the prediction to the last observation from the training data (the reference point). This chart highlights the impact of each feature or concept on shifting the forecast from the reference point to our target prediction. Users can track each factor's contribution, categorized as positive (in red) or negative (in gray). By following the arrows, users can see how different elements affect the model's output, aiding in informed decision-making. A higher importance score indicates a stronger link between a feature and the forecast difference. Hovering over the bars reveals more details on each factor's effect on the results.

For example, starting with the known reference value, the analysis from the explainer reveals that the primary factor influencing the trained model is yearly periodicity. Yearly periodicity, which refers to the recurring patterns observed each year, has contributed to an increase in the forecast. Following this, the data trend, indicating the general direction in which the data is moving over time, also plays a significant role. The time series shows an upward trend between the reference and the prediction, leading the model to predict an increase in the forecast. Consequently, the final prediction is higher than the reference value.

In [33]:
exp[0].to_dataframe(n_features=10)
[2024-08-15 19:17:52,129] [automlx.mlx] Displaying only non-zero attributions.
Out[33]:
Feature Attribution
0 Yearly periodicity of Finance_W142 70.394186
1 Trend of Finance_W142 18.605814
In [34]:
exp[0].show_in_notebook()
Out[34]:

The above explanation provided a comparative analysis of the specified forecast time point versus the default reference time point, which is automatically set to the last observation in the training data. However, advanced users can change the reference and set it to another observation (-13, the most recent peak, in this example) in the training data as shown below.

In [35]:
advanced_exp = explainer.explain_prediction(forecast_timepoints=[10], reference_timepoint=-13)
advanced_exp[0].show_in_notebook()
Out[35]:

References¶

  • Oracle AutoML: http://www.vldb.org/pvldb/vol13/p3166-yakovlev.pdf
  • sktime: https://www.sktime.org/en/latest/
  • statsmodels: https://www.statsmodels.org/stable/index.html
  • M4 Competition: https://mofc.unic.ac.cy/m4/
  • Airline Dataset: https://www.sktime.org/en/stable/api_reference/auto_generated/sktime.datasets.load_airline.html
In [ ]: